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Economists  and  other  users  of  statistical  methodology  often  posit  a 
probabilistic  model  of  some  real  world  phenomenon  which  has  more  unknown  param- 
eters than  there  are  sample  observations.   In  such  cases  it  is  usually  impossible 
to  jointly  estimate  all  parameters  from  the  sample  data.   Even  in  those  instances 
where  there  are  well  established  estimation  procedures  when  the  number  of  sample 
observations  n  is  larger  than  the  number  of  parameters  r,  these  methods  are 
generally  inadequate  when  n  <  r^  as  the  necessary  calculations  cannot  be  carried 
out.   Furthermore,  when  n  is  only  "slightly  larger"  than  r,  such  estimates  often 
prove  to  be  unreliable  in  more  than  one  sense. 

One  particular  example  of  such  a  problem  that  frequently  occurs  in  analysis 
of  psychological  and  of  economic  data  is  this:   the  researcher  posits  a  linear 
regression  model  as  defined  in  (2)  below  with  r-1  independent  variables  but 
only  n  <  r  observations  on  the  dependent  variable.   Since  the  standard  least 
squares  procedure  cannot  be  applied,  he  may  ask  the  following  seemingly  reason- 
able question:   "What  subset  of  the  r-1  independent  variables  should  I  select 
for  inclusion  in  a  "new"  model  to  which  I  can  apply  the  standard  least  squares 
procedure?" 

Our  purpose  here  is  to  demonstrate  that  one  frequently  used  ad  hoc  method 
for  determining  such  a  subset  by  ordering  simple  sample  correlation  coefficients 
can  be  highly  misleading.   Any  procedure  which  uses  a  given  set  of  sample  data 
to  both  determine  the  structure  of  the  model  to  be  tested  and  to  estimate  param- 
eters of  this  model  is  intuitively  unsettling.   Here  we  present  tables  which 
quantitatively  demonstrate  how  dangerous  such  ad  hoc  methods  can  be. 


Ad  Hoc  Use  of  Simple  Correlation  Coefficients  to  Determine  Model  Structure 


Consider  an  r-dimensional  Independent  Multinormal  process  defined  as  one 


that  generates  independent  r  x  1  random  vectors  x 
densities 
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We  wish  to  estimate  parameters  of  the  conditional  distribution  of  xj^  '    given 
x^   ,...,x^  '   when  neither  ^   nor  h  is  known  with  certainty  from  a  set  of  n 
vector  sample  observations  x  "^  ^x   ,...^x^"  .   Alternatively,  we  may  consider 
an  r-dimensional  Normal  Regression  process  defined  as  a  process  generating 


independent  scalar  random  variables  according  to  the  model 


;p>, 
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the  X.   s  are  known  numbers  which  in  general  may  vary  from  one  observation  to 

the  next,  and  the  e.s  are  independent  random  variables  with  identical  normal 
'  J 


fj^(e|0,  h)  =  (2n)" 
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Suppose  now  that  we  have  n  observations,  and  let 
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be  a  matrix  of  observations  of  "independent"  variables  together  with  an 
additional  column  of  I's.   If  the  rank  of  X  X  is  singular  the  following  method, 
or  a  slight  variation  of  it,  is  often  employed  to  make  standard  least  squares 
"work": 

1.  Using  the  n  sample  observations,  compute  r-1  sample  statistics 
p.,  i=2,...,r,  where  p.  is  the  simple  sample  correlation 
coefficient  between  x,  and  x. . 

2.  Relabel  variables  X2,...,x  so  that  jp-]  >  [po]  >  ...  >  |p  [• 

3.  Choose  an  integer  r*  <  rank  (X  X). 

4.  Restructure  the  model,  eliminating  from  consideration 

relabelled  variables  x  .,.,..., x  .   (Henceforth  we  call  the 
r*+l'    '    r 

initial  model,  "Model  A"  and  the  restructured  model,  "Model  B".) 

5.  Assuming  that  Model  B  is  an  adequate  approximation  to  Model  A 
use  X  X  to  estimate  parameters  of  the  conditional  distribution 
of  X,  given  the  values  of  relabelled  variables  X2,<...^x^^, 

3«   Simulation  Study 

Model  B  is  a  strongly  biased  approximation  to  Model  A,  and  the  magnitude 
of  the  bias  is  a  function  of  the  joint  distribution  of  the  r*-l  largest  simple 
sample  correlation  coefficients.   Analytical  expressions  for  this  distribution 
and  for  the  distribution  of  the  sample  multiple  correlation  coefficient  in 
Model  B  are  in  general  extremely  complex  and  unwieldy.   Hence  we  have  resorted 
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to  simulation!  in  order  to  numerically  determine  the  salient  features  of  Model  B. 
The  steps  in  this  simulation  were: 

1.  We  assumed  that: 

(a)  the  data  generating  process  is  a  Model  A  as  defined  in  (2)j 

(b)  the  dependent  variables  y*'-''^  and  the  independent  variables 
x!"  ,  > .  .  yX        ,    j=1^2^..,,n,  are  mutually  independent  random 
variables  identically  distributed  uniformly  on  [0^  1], 

2.  For  each  (n,  r)  pair,  n=10(2)20,  30,  40,  50,  and  r=10(5)50,  50(10)100, 
150,  200  we  generated  50  sets  of  n  observations  each. 

3«   For  each  set  of  n  observations  we  used  the  ad  hoc  procedure  out- 
lined in  section  2  to  calculate  the  five  largest  simple  sample 
correlation  coefficients  and  Model  B  multiple  correlation  coefficients 
for  Model  B's  with  m=2,  3,  4,  and  5  independent  variables. 

4.   Regarding  each  of  the  five  sets  of  50  simple  sample  correlation 

coefficients  as  a  set  of  sample  observations  we  then  calculated  the 
sample  meanjp.|  and  sample  variance  V(|p.|)  of  the  absolute  value  of 
each  of  the.  five  largest  simple  sample  correlation  coefficients.   The 
results  are  tabulated  in  Tables  1  and  2. 


'Ames  and  Reiter  [1]  performed  a  somewhat  similar  experiment  in  a 
different  context:   they  drew  100  economic  time  series  at  random  from  the 
Historical  Statistics  of  the  United  States  and  then  computed  the  sampling 
distributions  of  correlation  and  autocorrelation  coefficients  of  the  series 
drawn,  finding  that  "correlations  and  lagged  cross-correlations  are  quite  high 
for  all  classes  of  data.   E.g.,  given  a  randomly  selected  series,  it  is  possible 
to  find,  by  random  drawing,  another  series  which  explains  at  least  50  per  cent 
of  the  variances  of  the  first  one,  in  from  2  to  6  random  trials,  depending 
on  the  class  of  data  involved."  [1],  p.  637. 


5.  We  follow  a  similar  procedure  with  the  sample  multiple  correlation 
coefficients.   In  Table  3  the  mean  R  of  the  square  of  50  sample 
multiple  correlation  coefficients  for  model  B's  with  a  constant 
term  plus  m=2,  3,   k,    and  5  independent  variables  are  tabulated. 
The  tables  given  us  a  reasonably  accurate  ideal  of  the  order  of  magnitude 
of  Model  B  sample  correlation  coefficients  and  of  the  five  largest  simple  sample 
correlation  coefficients  whatever  the  distribution  of  the  independent  variables-- 
so  long  as  they  are  mutually  independent  and  have  finite  mean  and  variance. Tl 
Tables  2  and  4  of  the  sample  variances  of  the  |p . |  and  of  sample  multiple 
correlation  coefficients  show  that  variations  in  Tables  1  and  3  entries  due 
to  sampling  error  are  practically  negligible;  e.g.  the  coefficient  of  variation 
of  Ip-,1  for  n=  10  and  r=  10  is  less  than  .001  as  calculated  from  Tables  1  and  2. 

Evaluation 

Examination  of  Tables  1  and  3  illustrate  clearly  that  not  only  is  the 
procedure  outlined  in  section  2  biased,  but  that  bias  of  an  order  of  magnitude 
shown  in  Tables  1  and  3  entries  will  occur  with  extremely  high  probability.   For 


'In  order  to  assess  the  effect  of  assuming  that  the  data  generating  process 
generates  values  of  mutually  independent  but  uniformly  distributed  random 
variables,  we  duplicated  the  experiment  for  n=    and  r=  10  under  a  different 
assumption:   each  sample  observation  was  generated  by  summing  10  values  of  mutually 
independent,  identically  uniformly  distributed  random  variables.   This  spot  check 
using  an  approximately  normal  data  generating  process  revealed  no  differences  to 
the  third  significant  digit  between  values  generated  under  this  assumption  and 
values  tabled  in  Tables  1,  2,  3,  and  4. 

_1_L  _   2 

''One  interesting  feature  of  Table  1  is  that  for  given  r  and  i,  |p. | 
decreases  almost  exactly  proportional  to  1/n.   This  is  most  pronounced  for 
i=2. 
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example^  for  n=10^  r=10^  |p^2l  =  "61^  with  sample  variance  of  only  „0172  and  for 
n=     f    ^=  }    \po\-  with  sample  variance  of  only       even  when  the 

population  values  of  |p.|  i=2,...,r  are  controlled  to  be  01 

One  effective  way  of  dealing  with  the  problem  is  to  reformulate  it  in 
Bayesian  terms  so  that  we  may  systematically  incorporate  information  the  researcher 
possesses  from  sources  other  than  the  sample  into  the  analysis.   In  [2]  we  indicate 
how  Bayesian  estimation  of  the  mean  vector  and  variance-covariance  matrix  of  a 
multinormal  process  may  be  done  even  when  the  dimensionality  of  the  process  is  r 
and  there  are  only  n  <  r  sample  observations  available.  And  in  [3]  we  show  how 
Bayesian  estimation  can  be  done  when  the  data  generating  process  is  one  frequently 
occurring  in  econometric  analysis--a  set  of  simultaneous  linear  equations  with 
stochastic  components--and  there  are  less  vector  observations  than  parameters. 


[1]  Ames^E.  and  Reiter  S.;,  "Distributions  of  Correlation  Coefficients  in  Economic 
Time  Series,"  Journal  of  the  American  Statistical  Association,  56  (1961), 
pp.  637-656, 
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Institute  of  Technology,  Alfred  P«  Sloan  School  of  Management,  Working 
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